Moving object detection is the essential process of object recognition, marking and tracking in video sequences, the background subtraction algorithm is widely used in moving object detection. Concerning the problem that illumination changing, noise and local motion seriously affect the accuracy of moving object detection, a moving object detection algorithm based on background subtraction for video sequences was proposed. The background subtraction was combined with inter-frame difference to estimate the motion state of current frame pixels. The related pixels in the static and motion region were replaced and updated respectively. The Otsu method was used to extract moving object and the mathematical morphological operation was used to eliminate the noise and redundant information in the objects. The experimental results show that the proposed algorithm has good visual effect and high accuracy for detecting moving objects in video sequences, and it can overcome the shortcomings such as local movement and noise.